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1.0 EXECUTIVE SUMMARY 

1.1 BACKGROUND 

For most Department of Defense (DoD) installations, the prime method of detennining 
compliance with air regulations (and usually the primary backup for in-stack instruments) relies 
on trained observers to visually estimate the opacity of a plume every 15 seconds for a specified 
period, then average those estimates to detennine a single reading that is compared to pennitted 
levels. This is known as Environmental Protection Agency (EPA) reference Method #9 (40 
Code of Federal Regulations [CFR] 60, Appendix A, Method #9). [11] The sampling period 
may be as short as 6 minutes or as long as 1 hour. To qualify as a visual observer, an individual 
must attend a “smoke school” every 6 months. A smoke school consists of both classroom and 
field training, with a series of visual observations of a calibrated smoke source, constituting the 
qualification examination. That examination requires that the observer estimate the opacity of 25 
white and 25 black smoke plumes within ±15% accuracy for an individual observation and be 
within 7.5% overall. Sample opacity ranges from 0 to 100% in 5% increments. 

While Method #9 has an extensive history of use, it is both subjective and expensive in terms of 
dedicated work force. In addition to the cost of the school (approximately $350 per student and 
per diem/travel as needed), the 5 days of missed work add up significantly when multiplied by 
the number of students — for example, 35 at Hill Air Force Base (AFB), 25 at Elmendorf AFB 
—and the twice-yearly training requirement. 

1.2 OBJECTIVES OF THE DEMONSTRATION 

The objective of the demonstration was to field a digital opacity compliance system (DOCS) to 
be usable as an alternative to certified EPA Method #9 observers. To accomplish this, a digital 
recording and analysis system for opacity compliance measurements was demonstrated and 
validated as an alternative to the current opacity measurement procedure (40 CFR 60, Appendix A, 
Method #9). The system uses a commercial-off-the-shelf (COTS) digital programmable camera 
to capture images of a smoke plume. These images are then downloaded to a standard personal 
computer where they are analyzed 1 to detennine opacity. DOCS is designed to produce opacity 
readings, which are accurate, and provide a permanent digital record of the image for future 
reference. Preliminary testing showed the system to be highly effective in quantifying smoke 
plume opacity and in determining the opacity of fugitive dust. For example, DOCS had been 
used to take preliminary, limited measurements at a visible emission-training program; this 
preliminary investigation showed that DOCS easily passed the requirements of precision and 
accuracy as established by Method #9. 

1.3 DEMONSTRATION RESULTS 

Demonstrations in this effort were conducted at three commercial smoke schools—Ogden, Utah; 
Augusta, Georgia; and Columbus, Ohio—to demonstrate the system’s ability to qualify as a 
smoke reader would. Also, demonstrations at commercial and government sites in Ohio and 


1 The analysis software is a proprietary product developed by Scientech, Inc. Hill AFB is presently pursuing patent 
rights for all of DoD. 
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Alaska were conducted to evaluate quantitatively how the system performs compared to Method 
#9 observers in the field. Site measurements were necessarily confined to the < 40% opacity that 
is allowed by existing pennits for most measurements with some excursion measurements at 
start-up. It was a good test of the suitability of the technology for use in the regulatory 
environment that exists currently rather than a test of its ability to replicate observers’ readings 
over the complete range of 0 to 100% opacity. 

1.4 REGULATORY DRIVERS 

The EPA Emission Measurement Center (EMC) in Research Triangle Park, North Carolina was 
a partner in this study because the ultimate goal of the effort was to receive regulatory 
acceptance of the newer technology as an alternative to Method #9. The data gathering and 
analysis was closely coordinated with the EMC to ensure data integrity and sufficiency for their 
purposes. 

1.5 DEMONSTRATION RESULTS 

The DOCS image acquisition and processing system was demonstrated in both the relatively 
controlled environment of a smoke school and at industrial sites. Results indicated that the 
digital camera system can replace certified smoke readers in instances where contrast is expected 
to be good, where weather is most often clear, and where expected opacities are low. When 
opacities are regulated at 0%, the system is very attractive because of its forensic record of the 
plume. Technicians at Hill AFB have been using the system since November 1999. They and 
State of Utah air quality regulators are pleased with the results and are committed to future use of 
the methodology. 

DOCS is more costly to install than training Method #9 certified observers, but a comparison of 
the annual operating costs showed an annual saving of $8,090 using DOCS versus certified 
readers at a large urban facility and an annual saving of $14,890 at a remote facility. In both 
cases, the purchase cost of a DOCS system, which is continually trending downwards, would 
easily be paid back within a year. 

1.6 STAKEHOLDER/END-USER ISSUES 

General 

The use of digital opacity measurements will assist primary end users such as Hill AFB, Utah; 
Wright Patterson AFB, Ohio; Fort Stewart, Georgia; and Eielson AFB, Arkansas, in 
demonstrating compliance with emission standards. Each base/facility will benefit to an extent, 
depending on the level of compliance mandated by the size and mission of the facility. 
Regulators’ air pollution control programs will benefit because the records of visible emissions 
will be better documented. This technology has far-reaching potential and could be used by 
many agencies currently attempting to comply with opacity requirements via EPA Method #9, 
particularly if the permitted opacity is very low, as is most often the case. 
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EPA Perspective 

At the request of the EPA’s EMC, a meeting was held in August 2003 at Research Triangle Park 
to discuss the results of the Environmental Security Technology Certification Program (ESTCP) 
tests and their implications in the use of the DOCS as an alternative to Method #9. Present at the 
meeting were EPA personnel from Research Triangle Park and EPA Regions 4, 6, and 8; and 
representatives from U.S. Army and U. S. Air Force facilities in Florida, Kentucky, Tennessee, 
Texas, and Utah. The following is a summary of the meeting discussions and conclusions: 

• For the present time, Method #9 must remain as the baseline against which digital 
imaging technologies are compared. The goal is to develop a separate reference method 
for digital imaging opacity measurements. 

• It is important to note that the DOCS measurement capabilities are a result of field 
testing. This process has defined what can be done using digital imaging at the present 
time. Other technologies will have to meet this same level of performance. 

• The performance specification developed by the group will define the operational 
requirements of any future digital-imaging-based opacity determination system. 

• The standard will specify that comparison is between the system under consideration and 
a transmissometer reading (i.e., the transmissometer remains the standard against which 
opacity readings are made). 

• Separate consideration should be given to the camera and the software used for analysis. 

• A set of standard photographs would be used for the performance comparison testing. 

• Method #9 will still have to be used for any enforcement actions. 

• There is considerable interest in using digital imaging for the analysis of fugitive 
emissions. Several group efforts will be carried out to explore development options, 
including ESTCP or individual DoD component funding. 

• Proceed with the development of tests in Utah (and perhaps Arkansas, Texas, Tennessee, 
and Kentucky) where the facilities’ Title V permits currently include the option of 
utilizing the DOCS technology as a valid opacity screening tool. Ensure that definite 
measurement objectives are set up so tests can be objectively evaluated. 

• Include a recommendation to extend the DOCS to fugitive emissions in the final ESTCP 
report. 

• Efforts to include digital opacity measurements in State Implementation Plans must 
ensure that no new stringency requirements (either more or less) are imposed. 

• Take measured steps with a DOCS approach, build confidence, and show consistency. 
This will open the door for future acceptance and use as a separate standard. 
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Clearly, the EPA’s EMC is strongly supportive of the potential use of digital imagery as a 
method for quantifying visual opacity. To provide federal and state regulators with a clear 
understanding of the conditions under which digital imagery may be a technically superior 
and/or scientifically equivalent compliance methodology, EMC has endorsed the following 
activities: 

• Proceed with a year-long regulatory pilot test of the DOCS technology in selected 
states—namely Arkansas, Texas, and Utah—with the full involvement of their respective 
EPA regional offices and state air quality compliance personnel. The purpose of these 
tests would be to demonstrate the use of DOCS as a screening tool for opacity 
measurements required under existing Title V air permits. Photographs taken would be 
cataloged and analyzed according to the protocols established under the ESTCP project. 
EMC personnel feel that positive results from such tests will help provide technical and 
programmatic support they need to build a case for the development of a fonnal 
alternative to Method #9. (The tests began in January 2004.) 

• Use a subset of the photographs obtained under the ESTCP DOCS field tests to produce a 
test protocol for any digital imaging approach to opacity measurements. These 
photographs would be tied directly to smoke generator transmissometer opacity values 
and would, therefore, be an acceptable standard for validating new digital-imaging-based 
opacity measurement systems. EMC has continued to stress the importance of 
establishing an EPA-approved test protocol for verifying the performance of all future 
digital imagery technologies. 

• There is substantial interest at EPA and within state regulatory agencies in using digital 
imaging technology for fugitive emissions. Since the DOCS software package consists 
of computer algorithms specifically designed to estimate the opacity associated with 
fugitive emissions, EMC has indicated its willingness to support further testing and 
development of this capability. 

Air Force Perspective 

There are several key points to be made: 

• The field testing program, as carried out under the ESTCP project with close EPA 
supervision, demonstrated that the DOCS could, in fact, make accurate opacity 
measurements under proper conditions (i.e., clear weather with good contrast conditions 
and opacities <40%). 

• Generally, as weather conditions grow worse (i.e., increased cloud cover, overcast skies, 
or stonny skies), both Method #9 certified readers and the DOCS opacity readings 
demonstrated decreased accuracy in measuring visual opacity. 

• The ability of the DOCS to provide a pennanent record of opacity for a given set of 
measurement conditions is a significant advantage over the subjective record of a 
certified human observer. 
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• There are valid technical reasons to expect that known software deficiencies (e.g., 
problems with cloudy or overcast skies, improvements in the graphical user interface, and 
opacity readings >60%) can be corrected or substantially improved through carefully 
targeted additional development. 

Recommendations 


• ESTCP provides additional funds for DOCS software enhancements as well as DOCS 
regulatory compliance verification activities. Specific issues to be addressed with 
supplemental funding support would include: (1) refinement of the DOCS opacity 
algorithms to improve their ability to quantify visible opacity under variable sky 
conditions, (2) support of the 1-year DOCS regulatory pilot testing activities currently 
being conducted at Elmendorf AFB, Arkansas; Hill AFB, Utah, and Fort Hood, Texas, as 
well as commercial facilities in Utah, and (3) development a technology evaluation 
protocol for documenting the perfonnance of digital imagery technologies under EPA- 
specified test conditions. Estimated amount required is $100,000. 

• The EMC should take the digital opacity test protocol developed under this project and 
certify DOCS—to the set of conditions under which it meets Method #9 requirements— 
as an alternative to Method #9. Alternatively, the test protocol could be developed into 
an entirely separate test method for digital-imaging-based opacity measurements. Either 
way, a definitive roadmap should be forthcoming for the establishment of another opacity 
measurement protocol. 
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2.0 TECHNOLOGY DESCRIPTION 


2.1 TECHNOLOGY DEVELOPMENT AND APPLICATION 


The equipment tested consisted of a commercially available digital camera, specifically a Kodak 
290 zoom digital camera with programmable features. The data card from the camera was 
analyzed using proprietary software to produce opacity readings using a Pentium II laptop or 
equivalent with Windows. See Figure 1. 


Dust Processing 
Displayed 



DC265/290 Digital Commercial Computer 

Camera With Proprietary Software 

Determines Opacity and 
Displays Results 





Method 9 Results 
Displayed. Print or Store 
on Disk 


Figure 1. Schematic of DOCS. 


The use of digital image processing is novel in its approach to determining opacity for regulatory 
compliance. It makes use of commercially developed image capture technology for sampling the 
data on site. Other than the human observer, the only competing technology in use today 
requires sampling from in-stack fixed monitors. Such systems have no portability and are costly 
to purchase and maintain. Digital image processing, on the other hand, is portable and the 
analysis can be done either on site, using a standard notebook computer, or later in the office 
using a common desktop system. The images/samples can be archived by storing them on 
computer media, and analysis can be re-accomplished at any time, an advantage obviously 
denied to the human observer. 


2.2 PREVIOUS TESTING OF THE TECHNOLOGY 


The subject digital image acquisition and processing system has been preliminarily demonstrated 
in the relatively controlled environment of a smoke school and at some industrial sites, both for 
smoke plumes and for quantifying fugitive dust. Technicians at Hill AFB have been using the 
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system since November 1999. With the encouraging results, the state of Utah has incorporated 
the option of using the DOCS for quantifying visible opacity within Hill AFB’s Title V permit. 

The system’s development and funding history prior to this field test is as follows: 

• SCIENTECH Corporation sponsored a “quick look”, feasibility study, providing $10,000 
to the Space Dynamics Laboratory (SDL), a nonprofit research institute at Utah State 
University. 

• Concept development and demonstration was funded through the National Aeronautical 
and Space Administration (NASA) Stennis Research Center’s Affiliated Research Center 
Program. NASA funds were provided directly to the SDL to support both staff and 
graduate student involvement for an 8-month effort. Estimated expenditure was $80,000. 

• Prototype development and demonstration of algorithms to allow for quantification of 
fugitive dust were sponsored by the Air Force under 3600 program funding provided by 
the Air Force Research Laboratory at Tyndall AFB, Florida. The value of that contract 
was $32,500, for which Hill AFB received one camera set and four software licenses. 

• SCIENTECH Corporation has invested approximately $25,000 in completing 
development of the analysis algorithms and finishing the graphic user interface and report 
generator. 

• The state of Utah’s Division of Air Quality purchased one system for field examination in 
February 2000 at a cost of $5,000 and Hill AFB purchased four systems in addition to 
their original site license for $8,000. Those funds have been re-invested into further 
system development. 

2.3 FACTORS AFFECTING COST AND PERFORMANCE 

Scientific reliability and forensic reproducibility are the biggest factors affecting cost and 
performance. The cost of maintaining certified smoke readers as per EPA Method #9 is 
considerable, but variable. Semiannual training will cost approximately $350 per student for the 
class. The additional costs for travel, per diem, and lost productivity are widely variable. Bases 
with large work forces, such as U.S. Army, Navy, and Air Force maintenance depots (e.g., Hill 
AFB) and other industrial/rework facilities can absorb lost productivity with existing manpower 
and some minor inconvenience, but smaller organizations, such as in the 611th Civil Engineer 
Squadron (CES) with primary environmental compliance responsibilities for remote U.S. Air 
Force facilities throughout Alaska have limited capability to absorb such costs. In these 
situations, twice yearly EPA-certified, contract replacement workers need to be transported to 
each site at the same time as the certified readers are transported off site for recertification. 
Establishing a DOCS system methodology, which requires one-time training, followed by online 
or CD-based retraining, could save considerable financial resources and enhance mission 
accomplishment by reducing these semiannual work force migrations for recertification. 
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2.4 ADVANTAGES AND LIMITATIONS OF THE TECHNOLOGY 


Advantages for this technology fall into two distinct categories. The first advantage is that it will 
save the DoD money. While the need for training is similar to smoke school training and will 
not be eliminated, the number of people who need to be trained will be cut nearly in half. Also, 
it is expected that they will need training only on familiarization with opacity measurements for 
one day per year instead of two or three days of smoke school training and qualification twice 
per year. The second advantage is in the forensic reliability and reproducibility of the 
measurements with the digital camera. Currently, Method #9 is a visual judgment of opacity; 
individuals record their observations on forms and may be asked to provide expert testimony in 
the future. The expert testimony then becomes merely an honest affirmation that the 
documented opacity is what the observer recorded on that day. The DOCS method will provide 
reproducible, reliable data. The digital photograph will represent an archive of the plume, as it 
existed at the time in question. 

Limitations of this technology are similar to that of a human observer. Current regulations 
governing the measurement of visual opacity place constraints on the angle of the observation 
with respect to the sun, and other meteorological parameters. Equivalency demonstrations have 
constrained this testing to those same parameters. Also, certified observers find it somewhat 
difficult to accurately discriminate plumes from background when sky conditions are overcast or 
when the background of the plume is complex. This technology has similar limits and, in fact, 
the system cannot adapt as well as humans to less than ideal field conditions (i.e., poor weather). 
The technology as it exists today is best used in good weather conditions and with homogeneous 
to low variability sky backgrounds. Also, opacity measurements with the digital system as 
presently developed should be limited to less than 40% opacity. This encompasses nearly all 
regulated plumes at this time. 

Finally, DoD facilities in Alaska, Texas, and Utah as well as EPA Regions 6 and 8 are 
conducting a 1-year DOCS regulatory pilot test during which both visual observations and digital 
photographs of emissions from regulated industrial processes are being evaluated during normal 
compliance inspection activities. For many of the permitted sources, a measured nonzero visible 
opacity constitutes a regulatory violation. The digital opacity compliance system with its 
forensic record is considered ideal for this purpose. 

Once approved by EMC as an acceptable method for estimating visible opacity, the procedure 
for allowing the incorporation of digital imagery technology into air quality permits will be 
through the appropriate modification of existing state implementation plans (SIP). Once in the 
SIP, regulated facilities would have the option of requesting the use of digital imagery to 
demonstrate compliance with visible opacity limits. 
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3.0 DEMONSTRATION DESIGN 


3.1 PERFORMANCE OBJECTIVES 

The objective of this technology demonstration was to determine whether the ability of the 
DOCS to measure opacity is equivalent to EPA Method #9. The EPA EMC was a participant in 
this test program which required that the digital measurement of smokes and computer 
computation of opacity be thoroughly explored. EMC also required that the system be able to 
measure opacity during smoke school with the same reliability as human observers who qualify 
for EPA Method #9 certification. Moreover, the EMC requires that the system successfully 
operate in the field at least over the range of opacities currently permitted for today’s sources. If 
equivalency were successfully demonstrated, DOCS would be considered an acceptable 
alternative to EPA Method #9 for verifying visible emission compliance. Table 1 describes the 
performance criteria and metrics used to characterize the equivalency of DOCS to EPA Method 
#9. 


Table 1. Performance Objectives. 


Type of 
Performance 

Primary Performance 

Expected Performance 

Actual Performance 

Objective 

Criteria 

Metric 

Objective Met? 

Quantitative 

1. Determine the statistical 

Demonstrate that the DOCS 

Partially 


equivalency of the 

opacity measurements are 



DOCS to EPA Method 

statistically within the EPA 



#9. 

Method #9 acceptable 
margin of error (i.e., ± 

7.5%) at the 99% 
confidence level. 

Partially 


2. Estimate reliability of 

No statistical difference 



the DOCS relative to 

between the variability at 



certified human 

99% confidence of the mean 



observers. 

difference in opacity 
readings reported by the 
DOCS and the variability of 
the mean opacity differences 
reported by recently 
certified smoke readers. 


Qualitative 

1. Reproducible results 

Included into above 

Yes 


2. Near real-time 

Increased number of 

Yes 


monitoring 

observations. 



3.2 SELECTING TEST SITES/FACILITIES 

The goal was to collect a wide range of opacity measurements representative of various types of 
smoke emissions. Sites were selected, with the concurrence of the EPA EMC, in different 
climatic regions of the United States where smoke school certification was being conducted and 
ultimately where regulated smoke stacks exist and there is a professed need for this technology. 
Locations representing the atmospheric conditions and climates of the Southeast, Midwest, and 
Western United States as well as Western and Central Alaska were all of concern to the EPA. 
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3.3 TEST SITE/FACILITY HISTORY/CHARACTERISTICS 


Sites for smoke schools included Columbus, Ohio; Augusta, Georgia; and Ogden, Utah. The 
EMC recommended attending smoke school with the DOCS system at the locations 
approximating those discussed in Section 3.2, using Eastern Technical Associates (ETA) of 
Gamer, North Carolina, as the preferred smoke school. ETA represents more than 80% of the 
smoke school business in the United States and there was an established, professional working 
relationship between ETA and EPA EMC. By using the test smoke generated by a smoke school 
opacity source, the DOCS was exposed to the same range of opacity challenges, and under a 
variety of environmental conditions as smoke school students are required to view (25 black and 
25 white smokes ranging from 0 to 100% opacity). The follow-on test site demonstrations then 
showed the feasibility of operating the equipment at practical real-world facilities under a variety 
of atmospheric conditions on the range of smoke-generated opacity that is seen and regulated in 
the field. It was important to the project’s advisory committee and to the EMC that the 
technology be tested under differing environmental conditions. However, the sites chosen also 
allowed for measurement of practical smokes from combustion of diesel fuel, natural gas, coal, 
miscellaneous trash, wood chips, and waste oils. 

3.3.1 Eastern Technical Associates 

ETA was founded in 1979 and currently is overwhelmingly the predominant U.S. provider of 
smoke school training in support of 40 CFR 60, Appendix A, Method #9. 

To certify as a qualified observer, a person must be tested and demonstrate the ability to 
accurately assign opacity readings in 5% increments to 25 random black plumes and 25 random 
white plumes. Error cannot exceed 15% opacity on any one reading and average error cannot 
exceed 7.5% opacity for each smoke category. The DOCS was required to demonstrate the same 
proficiency. 

3.3.2 Augusta, Georgia, Smoke School 

Fort Stewart is the largest Anny installation east of the Mississippi River and is home to the 3rd 
Mechanized Infantry Division. They expressed an interest in possibly using a digital system as 
backup of their stack monitoring system. The nearest smoke school to Fort Stewart was at 
Augusta, Georgia. Climatic conditions in Augusta are representative of those throughout the 
Southeast United States, which includes Fort Stewart and many other installations. 

3.3.3 Columbus, Ohio, Smoke School and Wright Patterson AFB, Ohio 

Wright-Patterson AFB (WPAFB) is the largest, most diverse and most complex base 
organizationally in the Air Force. The base is located near Dayton, Ohio and the nearest smoke 
school was in Columbus. WPAFB is home base for more than 60 units representing a host of 
Air Force and DoD organizations. It has a work force of approximately 24,600, with nearly 
15,500 being civilians, making it the fifth largest employer in Ohio and the largest employer at a 
single location. The 88th Air Base Wing Environmental Management Office is responsible for 
the environmental programs on base, including the management of all air permits. 
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3.3.4 Ogden, Utah 


Hill AFB, Utah located near Ogden, is home to the Ogden Air Logistics Center. The Hill AFB 
Environmental Management Directorate is responsible for the environmental programs at the 
base, manages all pennits, including air pennits, and sponsors a smoke school, which was held at 
Ogden. 


3.3.5 Facilities Within Alaska 

The EPA EMC indicated that several facilities in Alaska have been asking for technology to 
replace Method #9 and that HMH Consulting of Anchorage would be able to test the technology 
at various facilities within the state, including the following: 

• Anchorage Water and Wastewater Utilities smoke generator and sludge incinerator at the 
J.M. Asplund Wastewater Treatment Facility in Anchorage; 

• Golden Valley Healey Electric Association coal-fired power plant in Healy, 
approximately 110 miles southwest Fairbanks; 

• Alyeska Pipeline Service Company trash incinerator at Pump Station #7, which is near 
Milepost 400 on the pipeline outside Fairbanks; 

• Eielson Air Force Base coal-fired heating plant at Eielson AFB, approximately 20 miles 
southeast of Fairbanks; and 

• Williams North Pole Refinery in North Pole, approximately 15 miles southeast of 
Fairbanks. 

All sites were visited and the technology demonstrated during the first week of September 2002. 

3.4 PRESENT OPERATIONS 

Installations that generate their own steam or power or that have special purpose industrial 
operations, such as medical waste incinerators, hush houses, and small foundries are regulated 
with respect to the visible emissions from stationary smoke stacks. This regulation relies on 
EPA Reference Method #9, Visual Determination of the Opacity of Emissions from Stationary 
Sources [1,2]. 

Method #9 requires trained observers to determine compliance by visually estimating the opacity 
of the smoke from sources. The level of regulation, i.e., the opacity allowed for the specific 
source, can vary usually between 0 and 40%, depending on the location of the source and the 
perceived impact to the environment the source represents as detennined by the local, state, or 
federal regulators. Qualified observers are trained in accordance with EPA Reference Method 
#9, Section 3. 


13 



3.5 PRE-DEMONSTRATION TESTING AND ANALYSIS 


The technology had undergone a number of successful preliminary field demonstrations at Hill 
AFB as well as at other locations in the United States. In addition, a project scientific advisory 
panel met frequently to review the basic approach involved in the opacity measurements to 
ensure that testing was fundamentally sound and based on accepted methods, which are in 
general use. The panel also assisted in developing protocols to ensure that equivalency with 
Method #9 was achieved. 

3.6 TEST AND EVALUATION PLAN 

3.6.1 Demonstration Setup and Start-Up 

The equipment was hand carried to each smoke school location or industrial facility. Each 
demonstration was conducted as prescribed in EPA Method #9, Appendix C, and in standard 
operating procedures (SOP) (refer to Appendix E of the Final Report [3]). A certified smoke 
generator was operated specifically for the purposes of testing the DOCS system during Phase I 
smoke school demonstrations. These measurements were accomplished as part of a certified 
smoke school. SOPs were followed at smoke schools and in field validations and they provided 
for technology maintenance and for operations in the event of unforeseen problems. 

3.6.2 Period of Operation 

3.6.2.1 Phase I Operations 

EPA-approved smoke schools lasted for a total of three days at each site. The DOCS 
demonstration was run continuously during the duration of the smoke school. 

3.6.2.2 Phase II Operations 

Phase II field demonstrations were conducted, as much as possible, to encompass a 
representative sample of all the different conditions experienced during a complete day’s 
operation at each field site. Operating hours were roughly 0700-1800 hours, adjusted to ensure 
samples were taken during daylight hours. 

The Anchorage Water and Wastewater Sludge Incinerator. At this field demonstration site, 
smoke readings were taken in conditions thought to be challenging to the software, i.e. reading 
light colored smoke against a gray background sky. However, the opacities experienced were 
relatively low and, therefore, would have been adequately handled by the software. The plume 
was generated by incineration of municipal water and wastewater treatment sludge. 

The City of Anchorage Public Works Facility Smoke Generator Test Facility. This field 
demonstration site provided a unique opportunity for testing the DOCS system on the full range 
of opacities in both black and white smoke. Six smoke-school-type testing runs were monitored 
with four camera systems and seven certified readers reading in parallel (>1,500 data points). 
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The Golden Valley Electric Power Station #1. This site provided an opportunity to measure soot 
from coal-fired combustor systems (black smoke) against a variable background. The six runs 
captured there, although on the low end of opacities, provided a measure of practical use for the 
camera system in the field. 

The Alyeska Pipeline Pump Station # 7. This trash incinerator was barely able to generate visible 
smoke and was unable to sustain opacity greater than 20% for more than a minute or two. 
However, the tests there provided the project with a broad variety of opacities from the facility 
testing the camera system’s ability to capture quick and dramatic changes in opacity alongside 
smoke readers. 

The Eielson Coal-Fired Boiler Plant. This was another facility that found it difficult to generate 
visible smoke for more than very short periods of time. The variety of opacities and variability 
over short periods of time were a stiff test of the digital system. Of concern here was the 
potential for reading smoke during winter months when smoke, steam, no sunlight, and a gray 
background as well as very low temperatures typically below -40 °F made conventionally 
reading smoke difficult. 

The Williams North Pole Refinery. People at this facility were very helpful to the team and 
provided as much variability in opacity as they were capable of producing. Although it was not 
possible to obtain the high opacities that were thought to be possible, the data were a good test of 
the system at levels of opacity that would be encountered in routine use. These winter 
measurements added to the overall data. 

3.6.3 Experimental Design 

The field demonstration program in both phases strictly followed EPA Method #9 for certified 
smoke readers, and the design of the DOCS demonstration was as stated in the DOCS SOPs. 
Phase I qualification testing was conducted in the same way as for smoke school attendees with 
the transmissometer results being the standard of comparison for both the students and the 
DOCS. The Data Quality Assessment Process is described in Appendix B of the Final Report 
[3]. Statistical comparisons (see Table 7 in Section 4.3.3) were made between the DOCS and 
students qualifying on any particular run. 

The second phase of the field demonstration operated the DOCS system in a real-world 
environment subject to the same conditional parameters as stated in EPA Reference Method #9. 
Measurements were made of the six sources chosen at specific locations described in Section 
3.6.2.2 per direction from the advisory committee and EPA EMC. Each field validation test site 
either had continuous opacity measurement devices available and certified smoke readers as a 
backup or used smoke readers as a primary opacity measurement method. The sites represented 
different sources of smoke under varied circumstances and therefore produced different types of 
smoke plumes. Although the DOCS SOPs and EPA Method #9 prescribe that the DOCS and 
certified smoke readers from the units being tested should be given an opportunity to “calibrate” 
against generated smoke of known opacity (contracted from a smoke school) prior to observing 
and detennining opacity for real plumes, this was not possible in the field. 
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Expert Panel: An expert panel was assembled of select members from academic, EMC, EPA 
Regions, the U.S. Air Force, and other governmental sources who had demonstrated expertise in 
digital computer-based algorithm processing, measurement of opacity and smokes, visibility 
determination and modeling, or statistical experimental design. This panel reviewed the 
scientific basis of digital opacity measurements to provide EMC with the basis for 
recommending an alternative to Reference Method #9 in the Federal Register. The panel 
reviewed the data quality objectives (see Appendix B of the Final Report [3]) for both the smoke 
school and field test measurements and developed specifications/parameters within which the 
field work was conducted for EPA acceptance. 

3.7 DEMONSTRATION SCHEDULE 

3.7.1 Phase One 

Smoke School Demonstrations: First year fieldwork during late 2001 and early 2002 involved 
measurements of opacity concurrent with established smoke schools in Ogden (September 2001), 
Augusta (December 2001), and Columbus (March 2002). These field measurements were 
designed to demonstrate the ability of the digital opacity system to qualify as a certified smoke 
reader over the complete range of test smokes, i.e. 0% to 100% opacity. 

3.7.2 Phase Two 

Site Demonstrations: The objective of the Phase Two measurements was to demonstrate the 
technology’s ability to generate actual measurements in the field, i.e., over the fairly narrow 
range of pennitted opacities under various environmental conditions. Measurements were made 
concurrent with trained observers having variable levels of residual calibration (since they would 
not be freshly trained via smoke school.) The DOCS was field-tested at each of the following 
sites: 

• Anchorage Water and Wastewater Sludge Incinerator - September 5, 2002 

• City of Anchorage Public Works Facility Smoke Generator Test Facility - September 6, 
2002 

• Healy Golden Valley Electric Power Station #1 - September 9, 2002 

• Alyeska Pipeline Pump Station # 7 Trash Incinerator - September 10, 2002 

• Eielson AFB Coal Fired Boiler Plant - September 11, 2002 

• Williams North Pole Refinery - September 13, 2002 

Like Phase I, four COTS digital cameras (Kodak DC290 or Kodak DC265) were used to 
photograph visible emissions generated during the Phase II field tests. The DOCS commercial 
digital imaging software was installed and tested on each of the cameras before any photographs 
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were taken. No technical adjustments or physical modifications of the cameras were necessary 
to operate the DOCS camera software. 

Rather than having fixed positions relative to the sun, the DOCS camera operators (four in all) 
were allowed to position themselves in any valid Method #9 location relative to the stack. In 
addition to the four DOCS camera operators, from six to eight Method #9-certified readers were 
available at each industrial site to estimate plume opacity. The certified readers consisted of 
facility personnel as well as members of the DOCS technical evaluation team. 
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4.0 PERFORMANCE ASSESSMENT 


Under clear weather conditions, the performance objectives were met in the 0-40% range of 
opacity. At higher opacities, the performance of the DOCS system was more variable. When 
sky was employed as background, DOCS achieved Method #9 performance criteria, was at least 
as accurate, and was more reliable than certified smoke readers. However, under less than 
optimal sky conditions, human observers, who can employ background other than sky to 
improve the visual contrast of the plume, performed better. 

DOCS opacity measurements were reproducible and could be made in near real-time if the 
photographic images were downloaded to a personal computer and evaluated in the field using 
the proprietary software. 

More observations were possible with the DOCS system compared to using certified human 
observers because one human observer was able to capture and evaluate several DOCS images in 
a given time period compared to making a single human observation of a plume in the same time 
period. 

4.1 PERFORMANCE DATA 

4.1.1 Phase I Performance Data 

Approximately 6,400 opacity readings of visible emissions were taken as part of the DOCS field 
evaluation in Utah. Of that number, 4,741 opacity readings (2,336 black plumes and 2,405 white 
plumes) or nearly 75% of those computed were deemed acceptable for DOCS statistical analysis 
(see Table 2). The decision to exclude 2,161 opacity readings from the DOCS statistical 
analyses was based on technical problems that included: (1) physical obstruction of the smoke 
plume (e.g., trees, clouds, telephone poles, etc.), (2) folding, twisting, or other significant 
physical disruptions to the plume; and (3) inappropriate modification of digital image through 
use of the “brush” function that was available in the DOCS software package. 

The weather conditions during the smoke school field tests in Augusta were characterized by 
scattered clouds and partly overcast skies. Not only were the weather conditions in Augusta 
found to be appreciably different from those encountered in Ogden, but the physical landscape of 
the two smoke school locations were drastically different as well. For example, while the Ogden 
smoke school field tests were conducted in a large open parking lot located adjacent to a 
university athletic stadium, the venue for the Georgia smoke school was smaller and more 
secluded in a parking lot surrounded by a dense pine forest. The combination of cloudy 
conditions and tall trees provided a more variable background against which the DOCS and 
certified human observers were estimating plume opacity. 

Unlike the Utah smoke school field test, in the analysis of the Georgia smoke school DOCS data, 
the brush function was not used to analyze any of the digital photographs. Furthennore, several 
DOCS camera operators were certified as EPA Reference Method #9 visual observers so a larger 
number of EPA Reference Method #9-certified readers were available to participate in the side- 
by-side opacity method comparison during the first day of smoke school. 
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Weather conditions during the Columbus field tests were considerably different from either the 
Utah or Georgia smoke schools. The climate in Columbus for the field tests was characterized as 
near freezing temperatures, light rain mixed with snow, and thick, overcast skies. 


Table 2. Statistical Data Summary of Ogden, Utah Smoke School DOCS Evaluation. 


Color of Smoke—Opacity 
Measurement Approach 

Opacity 

Range 

Mean 

Deviation (%) 

Number of 
Samples 

95% Cl 1 
(%) 

99% Cl 
(%) 

Black—DOCS 

0-100% 

6.4 

2,336 

6.1-6.7 

6.0-6.8 

Black—certified observers 

0-100 

6.7 

246 

5.7-7.8 

5.4-8.1 







Black—DOCS 

0-60% 

5.6 

1957 

5.3-5.9 

5.2-6.0 

Black—certified observers 

0-60% 

5.4 

212 

5.5-6.4 

4.1-6.8 







Black—DOCS 

0-40% 

5.4 

1745 

5.0-5.7 

4.9-5.8 

Black—certified observers 

0-40% 

4.8 

194 

3.8-5.7 

3.5-6.1 







White—DOCS 

0-100% 

10.0 

2405 

9.5-10.5 

9.4-10.6 

White—certified observers 

0-100 

8.5 

282 

7.5-9.6 

7.1-10.0 







White—DOCS 

0-60% 

6.7 

1897 

6.3-7.0 

6 . 2-12 

White—certified observers 

0-60% 

8.2 

224 

7.0-9.4 

6.6-9.8 







White—DOCS 

0M0% 

5.9 

1686 

5 . 5-62 

msm i 

White—certified observers 

0-40% 

7.4 

199 

6.1-8.7 

5.7-9.0 


Cl - confidence interval 


Like the Augusta smoke school, in the analysis of the Columbus smoke school DOCS data, the 
brush function was not used to evaluate any of the digital photographs. Furthermore, since all 
the DOCS camera operators were also certified as EPA Reference Method #9 visual observers, 
more certified readers were available to participate in the side-by-side opacity method 
comparison during the first day of smoke school as compared to the Utah smoke school. 

4.1.2 Phase II Performance Data 

The performance confirmation method used in this study essentially consisted of statistically 
verifying that the mean difference between the in-line transmissometer opacity readings 
estimated over the full range of opacity and the opacity readings recorded by the digital opacity 
compliance system is within the acceptable margin of error established for EPA Method #9 
certification, i.e. ± 7.5 %. To ensure that sufficient data were collected to support a final 
decision at the 99% confidence level, the EPA’s Data Quality Objectives (DQO) process 
(Reference #6) was applied to estimate the minimum number of samples necessary to limit the 
decision errors to an acceptable rate. This is reflected in Appendix B of the Final Report [3]. 

Preliminary application of the DQO process in developing a DOCS technology sampling plan 
resulted in an experimental design that consisted of taking at least 86 opacity measurements (43 
black smoke and 43 white smoke) at each location to maintain the rate of committing a decision 
error to no more than 1% (99% confidence level). Since each opacity EPA Method #9 
certification run consists of 50 opacity measurements (i.e., 25 black smoke readings and 25 white 
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smoke opacity readings), a minimum of at least 200 randomly generated opacity data points is 
collected each day (assuming two certification runs conducted in the morning and two in the 
afternoon) during the certification program. Collecting a much larger number of data points than 
that estimated by the DQO process ensured that the final decision was supported by strong 
statistical evidence. 

A secondary objective in the technology evaluation program was to compare the reliability of 
DOCS to consistently and accurately measure plume opacity relative to certified EPA Method #9 
smoke readers. Reliability testing provides potential users of the DOCS technology a means for 
predicting its performance under field conditions. Comparative reliability testing involved 
taking simultaneous opacity measurements by both human smoke readers and DOCS and 
estimating their respective statistical variability. The estimated magnitude of the statistical 
variability (as reflected in the breadth of the 99% confidence interval) for each measurement 
system (certified human observers and DOCS) is used as an indirect measurement of its 
reliability. For example, if the DOCS plume opacity measurements were found to exhibit a 
statistical variability that was significantly less than the variability estimated for certified smoke 
readers, the conclusion would be that DOCS is more reliable than certified smoke readers. 

Except for the EPA-approved smoke generator in Anchorage and the coal-fired power plant in 
Healy none of the smoke stacks evaluated were equipped with opacity monitors. Without the 
benefit of a reliable and regulatory approved opacity monitor, questions relevant to the accuracy 
of the DOCS could not be addressed at these sources. However, field data from stacks not 
equipped with opacity sensors could be used to determine whether the DOCS and EPA 
Reference Method #9 would yield significantly different results when applied simultaneously to 
evaluate the opacity of smoke plumes. Therefore, the opacity data collected in Alaska were 
analyzed differently based on whether the recorded smoke plume opacity was supported by the 
measurements made by a continuous emission monitor (CEM). 

For data collected using the EPA-approved smoke generator and at the coal-fired power plant 
(i.e., those industrial sites supported by a CEM), the ability of the DOCS to measure plume 
opacity was compared with the results obtained from EPA Method #9-certified human observers 
(Table 2). For data collected at the industrial sites, the mean opacity deviations together with the 
95% and 99% confidence intervals were provided. The mean opacity, which was computed from 
the digital photographs using the DOCS, was compared to the Method #9-certified human 
observer’s reading. Table 3 and Table 4 summarize the statistical results obtained from 
aggregating the results from both field tests. 

Table 3. Statistical Data Summary of Alaska (CEM Data versus DOCS 

and Certified Observers). 


Opacity Measurement 
Approach 

Opacity 

Range 

Mean Deviation 
(%) 

Number of 
Samples 

95% Cl 1 

99% Cl 

DOCS - CEM 

0-100% 

14.1 

215 

12.2-16.0 

11.6-16.6 

Certified observers - CEM 

0-100% 

6.0 

224 


5.2-6.8 


Cl = confidence internal 
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Table 4. Statistical Data Summary of Alaska (DOCS versus Certified Observers). 


Opacity Measurement 
Approach 

Opacity 

Range 

Mean Deviation 
(%) 

Number of 
Sampling Means 

95% Cl 1 

99% Cl 

DOCS - Certified observers 

0-100% 

10.5 

360 

9.3-11.8 

8.9-12.2 

DOCS - Certified observers 

0M0% 

5.0 

255 

4.0-6.0 

3.7-6.3 


Cl = confidence internal 


During both the smoke generator and coal-fired power plant field tests, Method #9-certified 
readers appeared to generate opacity readings that were more consistent with the CEM opacity 
readings. It should be noted that there was no attempt to verify that the CEMs installed in either 
the smoke generator or at the coal-fired power plant were generating accurate opacity readings. 
In any event, the observation that certified readers were able to record opacity measurements 
more consistent with the CEM readings than the DOCS was surprising but not unexpected since 
the DOCS perfonnance is sensitive to sky conditions. Moreover, since many of the Method #9 
human observers involved in the side-by-side study were actually full-time operators at the coal- 
fired power plant, their extensive historical experience in estimating opacity from a particular 
industrial stack may have provided them with a unique advantage in reading visual opacity 
compared to the DOCS photograph readers who were observing the stack emissions for the first 
time. Despite the fact that Method #9-certified readers appeared to generate more accurate 
opacity measurements, the inability to visually establish the sun’s position raises concerns 
regarding the validity of some of the field results. 

4.2 PERFORMANCE CRITERIA 

The goal for this field demonstration project was to demonstrate the DOCS technology in the 
field and detennine if this technology was a scientifically defensible alternative to EPA 
Reference Method #9 for opacity measurements from stationary sources. ESTCP’s criteria for 
determining the perfonnance of the technology with respect to the reference method were 
generated with the help and guidance of the EPA Emissions Measurement Center and are listed 
in Table 5. 


Table 5. ESTCP Performance Criteria. 


Performance Criteria 

Description 

Primary or Secondary 

Validate as a scientifically defensible 
alternative to EPA Method #9* 

Ensure that the DOCS opacity measurements are 
statistically within the acceptable margin of error 
established for EPA Method #9 certification at 
the 99% confidence level. 

Primary 

Validate forensic reliability of system** 

Process the DOCS and certified human observer 
opacity measurement data and compute the mean 
difference and the associated 99% confidence 
intervals. 

Primary 

Improve amount of data collected 

Show the DOCS can operate continuously with 
no degradation to output. 

Secondary 

Ease of use 

Show that one trained individual can observe 
multiple emission sources with one system. 

Secondary 


* This was the initial part of Phase I. 

** Reliability of human smoke readers was compared to reliability of the DOCS in terms of precision and accuracy of data obtained in both 
phases of demonstration. 
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Supporting a decision on whether any proposed technology for measuring visible opacity 
(including DOCS) can be considered scientifically equivalent to EPA Reference Method #9 
would require that the approach meet, at a minimum, the standard of accuracy established by the 
published method. EPA Reference Method #9 requires that individuals applying for certification 
as visual opacity observers be able to estimate the visible opacity of 50 variable smoke plumes 
with a margin of error of no greater than 7.5%. The EPA Reference Method #9 accuracy 
requirement provides a primary performance criterion against which the DOCS technology can 
be evaluated. To minimize the potential of drawing erroneous conclusions from the opacity field 
data, the statistical accuracy of the DOCS technology in measuring visible opacity was 
characterized at the 99% confidence level. 

Beyond the question of whether the DOCS technology achieves the accuracy criterion specified 
within the EPA Reference Method #9 approach, the reliability of the DOCS relative to certified 
EPA Reference Method #9 human observers represents a critical usability consideration for 
environmental regulators, DoD compliance personnel, the private sector, and the public. To 
characterize the DOCS reliability relative to certified EPA Reference Method #9 human 
observers, the mean opacity differences between the DOCS and the EPA-approved 
transmissometer was compared to the mean differences estimated between the certified EPA 
Reference Method #9 human observers and the EPA-approved transmissometer. The variability 
of the mean opacity differences (as expressed by their 99% confidence intervals) was calculated 
using opacity field data. If the variability in the DOCS performance was found to be equal to or 
less than the variability characteristic of the opacity readings reported by the EPA Method #9 
human observers, the conclusion drawn would be that the DOCS is as reliable as EPA Reference 
Method #9. 

To estimate both the mean difference and its associated 99% confidence interval, a paired t-test 
statistical approach was employed. In this approach, for each opacity level reported by 
transmissometer (CEM), a corresponding DOCS and a certified EPA Method #9 visual observer 
measurement are used to establish the paired differences. 

The 99% confidence interval that characterizes the mean differences between the CEM and 
DOCS opacity readings can be used to detennine whether the DOCS accuracy is equivalent to 
EPA Reference Method #9. For example, if the range of the 99% confidence interval contained 
values that were above 7.5% (or below -7.5%), the conclusion drawn from the data would be that 
the DOCS is not as accurate as EPA Reference Method #9. On the other hand, if the range of the 
99% confidence interval was 7.5% or less (or equivalently, above -7.5%), the data would support 
the conclusion that the DOCS was equivalent to EPA Reference Method #9 (from the standpoint 
of accuracy). Table 6 shows the performance metrics and methodologies for evaluating the 
opacity measurements. 
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Table 6. Expected Performance and Performance Confirmation Methods. 


Performance Criteria 

Expected Performance Metric 
(pre demo) 

Performance 
Confirmation Methods 

Actual 
(post demo) 

PRIMARY 

Validate as 

scientifically defensible 
alternative to EPA 
Method #9 

Ensure that the DOCS opacity 
measurements are statistically within 
the acceptable margin of error 
established for EPA Method #9 
certification at the 99% confidence 
level. 

Paired t-test 

99% confidence Interval 

11.6-16.6 
DOCS to 
CEM 

5.2-6.9 
Observers to 
CEM 

Validate forensic 
reliability of system 

Mean difference in opacity readings 
between DOCS system and EPA 
Method #9 certified smoke readers 
is zero at 99% confidence. 

Paired t-test 

99% confidence interval 

8.9-12.2 at 
0-100% 

3.7-6.3 at 
0M0% 


4.3 DATA ASSESSMENT 

4.3.1 Phase I Data Assessment 

Data from the Phase I field study indicated that the DOCS can accurately and reliably measure 
the opacity of smoke plumes when weather conditions provide optimum color contrast between 
plume and background. Under clear, blue-sky conditions, the DOCS was able to consistently 
meet the EPA Reference Method #9 performance standard and, over opacity ranges of regulatory 
importance (i.e., 0 to 40 % opacity), DOCS accuracy was significantly better than that achieved 
by Method# 9 certified human observers (shown Table 6). 

As climatic conditions deteriorated, the DOCS had difficulty in accurately measuring plume 
opacity. Under weather conditions characterized by dark overcast skies, the DOCS consistently 
failed to meet the EPA Reference Method #9 perfonnance standard. However, Method-#9- 
certified human readers also experienced difficulty in accurately quantifying opacity under these 
conditions. In situations where the emissions can be viewed against backgrounds other than sky, 
human observers have a clear advantage over the DOCS during overcast sky conditions. Method 
#9 certification smoke schools, where the smoke generators were equipped with stacks that are 
only 15 feet above the ground surface, provided such an environment since this relatively short 
stack height allowed human observers to make opacity measurements using objects other than 
sky (light poles, vegetation, buildings, etc.) as background. 

4.3.2 Phase II Data Assessment 

Several industrial sites were selected in Alaska to assess the performance of the DOCS system 
on real-world smoke stacks in the field. The opacity data collected in Alaska were analyzed 
differently depending on whether the stack was equipped with an in-stack CEM opacity sensor 
(Table 3) or not (Table 4). Testing on the stack fitted with a CEM (Table 3) showed that DOCS 
failed to perform as well as the certified observers over the 0-100% range due to poor weather 
conditions. However, Table 4 shows that at the lower end (0-40%) of the range, DOCS 
performance more closely matched that of certified observers. 
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During Phase II field tests, climatic conditions were consistently unfavorable for measuring 
plume opacity. Light rain and overcast skies characterized the Alaska field test conditions. 
Results generated from the Phase II statistical analysis were consistent with those obtained 
during Phase I, namely, when the apparent opacity decreased due to the reduced visual contrast, 
both Method #9-certified human observers and the DOCS were unable to consistently achieve 
the EPA Reference Method #9 accuracy standard. Testing on stacks without a CEM showed that 
the perfonnance of DOCS of certified observers was closer at lower operatives (Table 4). 

4.3.3 Hypothesis Testing 

The initial documentation of method equivalency was established by computing the confidence 
intervals about the mean difference or deviation between the DOCS and transmissometer opacity 
readings. Based on a recommendation received from the EPA EMC, a more direct but 
comparable statistical approach for evaluating method equivalency is to determine if the DOCS 
field data supports the rejection of a null hypothesis (H 0 ). In significance testing, a null 
hypothesis (H 0 ) is developed that will be assumed to be true in the absence of strong quantitative 
evidence to the contrary. The null hypothesis (H 0 ) for the present study may be stated as 
follows: the true mean difference between the transmissometer and the DOCS opacity 
measurement methods is greater than 7.5%. This statement reflects the assumption that, in the 
absence of strong quantitative data to the contrary, the two opacity measurement methods are not 
equivalent. (The reader should note that the 7.5% mean difference is the allowable opacity 
deviation for human observers specified in EPA Reference Method #9.) Similarly, the 
alternative hypothesis (H a ) may be constructed as follows: the true mean difference between the 
transmissometer and the DOCS opacity measurement method is equal to or less than 7.5”. The 
rationale for constructing the null and alternative hypothesis in this fashion is to shift the burden 
of proof for demonstrating EPA Method #9 equivalency to the strength of the DOCS field data. 
In other words, in the absence of field data that strongly supports the rejection of the null 
hypothesis, the conclusion drawn from the data will be that DOCS is not equivalent to EPA 
Reference Method #9. Alternatively, if the strength of the data is sufficient to reject the null 
hypothesis (acceptance of H a ), the conclusion drawn from this study will be that the DOCS is 
statistically equivalent to EPA Reference Method #9. 

Table 7 summarizes the statistical comparison between the mean DOCS opacity measurement 
and the mean opacity measurement recorded by the certified human observers at facilities where 
there was no CEM installed. Over the full range of opacity evaluated, on average, the difference 
between the DOCS and certified human observers was approximately 10.5 %. 

When the comparison was made using a limited opacity range of 0 to 40% (based on certified 
reader measurements), the mean difference between the DOCS and certified human observers 
decreased to 5.0%. These data indicate that the difference between the two methods of opacity 
determination (DOCS and EPA Reference Method #9) is significant and that, on average, the 
DOCS tends to read smoke plumes at higher opacity levels than certified human observers. It 
should be noted that the data in Table 7 only addresses the issue of whether or not the response 
of the DOCS is significantly different from the observations recorded by certified human 
observers. The data does not address the issue of measurement accuracy. 
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Table 7. Summary of Hypothesis Testing Performed at the 0.05 Significance Level (at 

Smoke School Locations). 


Smoke 

School 

Color of 
Smoke 

Opacity 

Range 

(%) 

Mean 

Difference 

(%) 

Number of 
Samples 

Significance 

Level 

(a) 

Critical 1 

t-value 

Test 

Statistic 

Rejection 2 of 
Null 

Hypothesis? 

| Utah | 


Black 

0-100 

6.4 

2,236 

0.05 

1.96 

-6.77 

Yes 

Black 

0-60 

5.6 

1,957 

0.05 

1.96 

-12.08 

Yes 

Black 

0^10 

5.4 

1,745 

0.05 

1.96 

-12.50 

Yes 

White 

0-00 

10.0 

2,405 

0.05 

1.96 

10.39 

No 

White 

0-60 

6.7 

1,897 

0.05 

1.96 

-4.35 

Yes 

White 

0-40 

5.9 

1,686 

0.05 

1.96 

-8.99 

Yes 

| Georgia | 


Black 

0-100 

8.6 

4,949 

0.05 

1.96 

8.06 

No 

Black 

0-60 

8.2 

3,620 

0.05 

1.96 

4.49 

No 

Black 

0^10 

7.9 

2,896 

0.05 

1.96 

2.22 

No 

White 

0-100 

13.2 

3,535 

0.05 

1.96 

19.93 

No 

White 

0-60 

8.5 

2,565 

0.05 

1.96 

4.72 

No 

White 

0-40 

7.2 

2,203 

0.05 

1.96 

-1.47 

Yes 

| Ohio | 


Black 

0-100 

10.9 

3,498 

0.05 

1.96 

16.83 

1 No 

Black 

0-60 

9.4 

3,066 

0.05 

1.96 

10.24 

No 

Black 

0^10 

8.1 

2,753 

0.05 

1.96 

3.53 

No 

White 

0-100 

21.6 

4,394 

0.05 

1.96 

46.81 

No 

White 

0-60 

15.0 

3,758 

0.05 

1.96 

36.32 

No 

White 

0-40 

12.3 

3,131 

0.05 

1.96 

28.67 

No 


'From standard statistical tables, for a = 0.05 and n> 120, t-critical is approximately 1.96. 

^Where the null hypothesis is rejected, the data indicate that DOCS is equivalent to EPA Reference Method #9. Conversely, if 
the null hypothesis is not rejected, the conclusion is that DOCS is not equivalent to EPA Reference Method #9. 


4.4 TECHNOLOGY ASSESSMENT 

Although the DOCS is limited to the use of sky conditions to establish plume background, an 
added feature that was not employed in the current study was the DOCS “brush” function. The 
brush function essentially allows the photograph analyzer the option of selecting any segment of 
the digital image as background. The analyzer then “brushes” the selected background on both 
sides of the plume. The DOCS computer algorithm then compares the contrast between the 
plume and the selected background to generate an opacity reading. The opacity science advisory 
board determined that use of the brush function could lead to claims of bias or otherwise 
negatively impact the objectivity of the study. However, in a technical sense, certified readers 
who employ background other than sky are achieving the same goal as the brush function. In 
other words, human observers attempt to locate a background that provides maximum contrast 
with the plume in order to estimate opacity. Clearly, use of the brush function would have 
significantly altered the DOCS reading, but whether its use introduces significant bias and, 
therefore, challenges the legitimacy of the analyses has not been fully resolved by the opacity 
science advisory board. 

A third circumstance accounts for a potential advantage to digitally measuring opacity. Many 
small to medium size sources such as emergency power supplies or generators are regulated at 
0 % opacity. It is considered plausible by many local regulators that a digital system could be 
used for these sources (perhaps 50% of regulated sources) under existing state implementation 
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plans requiring that the operators not necessarily be very extensively trained if at all. The value 
over existing Method #9 would be in the archived digital pictures, which are a good stand-alone 
forensic record for demonstrating the nonexistence of visible emissions at a specific time and 
place. In these instances, the cost, although undetenninable at this time, would be very small 
compared to the EPA Method #9 approach. 

Several western states including Alaska, Texas and Utah are currently applying the DOCS 
technology within their air emissions compliance inspection activities. Depending on the results 
from this activity, data collected from both DoD and commercial facilities within these states 
during the 1-year DOCS regulatory pilot study may be used as the technical and legal basis for 
EMC to press forward in formulating a new EPA-approved test method for visible opacity 
determination. 
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5.0 COST ASSESSMENT 


5.1 COST REPORTING 

Existing methodology for source compliance with the EPA Reference Method #9 consists of 
training two individuals every 6 months. The cost of the training is $350 per class plus 40 hours 
of lost work (3 days of training and 2 travel days) at $ 15/hr and approximately $1,100 travel 
costs (hotel @$300, per diem @$105, $700 average airline costs) shown in Table 8 and Table 9. 
It is estimated that there are more than 3,400 training slots across the DoD every year. Current 
training and lost productivity costs are estimated at $ 11 million per year. 

The digital technology compliance costs cited here (Table 8 and Table 9) are for an initial class 
similar to smoke schools costing $350 per class, but for only 32 hours of lost work (2 days of 
class and 2 days of travel) at the same labor and travel rates. Recurring semiannual cost for the 
digital refresher class would be $350 for the class but would not involve travel costs because the 
class would be done on computer. Lost labor time would be no more than 8 hours and may not 
impact productivity as workers could take the class from their work stations in most cases. 

Each DOCS system costs approximately $3,000 and has a projected life span of more than three 
years. (Note that this cost is continually decreasing.) 

Table 8. Initial Cost Elements Used in Cost Analysis 


| Conventional Method #9 Training 

Proposed Initial Digital Computer Training | 

Remote Site 

Urban Site 

Remote Site 

Urban Site 

$350/person training 

$350/person training 

$350/person training 

$350/person training 

Labor: $ 15/hr—40 hrs 

Labor: $ 15/hr—40 hrs 

Labor: $ 15/hr—32 hrs 

Labor: $ 15/hr—32 hrs 

2 ea readers/source 

2 ea readers/source 

2 ea readers/source 

2 ea readers/source 

$1,100 travel/student 

$1,100 travel/student 

$1,100 travel/student 

$1,100 travel/student 

$1,100 travel/replacement 


$1,100 travel/replacement 


Total: $6,300/source 

Total: $4,100/source 

Total: $6,060/source 

Total: $3,870/source 


Table 9. Refresher Cost Elements Used in Cost Analysis 


| Conventional Method #9 Refresher Training 

Proposed Digital Computer Refresher Training | 

Remote Site 

Urban Site 

Remote Site 

Urban Site 

$350/person training 

$350/person training 

$350/person training 

$350/person training 

Labor: $ 15/hr—40 hrs 

Labor: $ 15/hr—40hrs 

Labor: $ 15/hr—8 hrs 

Labor: $ 15/hr—8hrs 

2 ea readers/source 

2 ea readers/source 

2 ea readers/source 

2 ea readers/source 

$1,100 travel/student 

$1,100 travel/student 

$0 travel/student 

$0 travel/student 

$1,100 travel/replacement 


$0 travel/replacement 


Total: $12,600/source 
(Train every 6 months) 

Total: $8,200/source 
(Train every 6 months) 

Total: $1,880/source 
(Train every 6 months) 

Total: $1,880/source 
(Train every 6 months) 


Equipment purchases for use of the digital system are estimated in Table 10 and Table 11 at 
approximately $3,000 per source. This includes: 

• Digital programmable camera system: $750 

• Dedicated computer: $1,500 
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• Software and license: $500 

• Miscellaneous support materials: $250 

A digital system such as DOCS is most applicable for monitoring sources that are regulated 
below 40% opacity, particularly for monitoring sources regulated to produce no visible 
emissions. This represents more than 80% of the regulated sources currently. The methodology 
for training digital camera operators to monitor opacity is expected to be similar to smoke school 
initially since there is a need to leam about smoke emissions and the laws being applied. The 
field work is not expected to take as long (32 hours instead of 40 for traditional smoke school) 
because the individuals will not need the practical exercises associated with calibrating 
themselves to read smoke. All other initial costs will be similar to the traditional smoke school 
(Table 8). The recurring training, i.e. the semiannual refresher training will be where the cost 
savings are incurred as there is an expectation that this training would be provided online, if it is 
required at all. Tuition costs are estimated to be the same as for traditional smoke school, but 
savings will be incurred in that there will be no travel expenses and the labor time will be 
reduced considerably (Table 9). 

Cost savings in using the digital method for opacity monitoring is expected to vary by the size of 
the facility or installation doing the monitoring. Large urban sources are impacted by the need to 
monitor emissions for opacity to a lesser extent than the smaller, remote sites. At large sites 
individuals are sent to smoke school and the lost labor is absorbed by redundancies in the 
relatively larger work forces than at smaller sites. Remote sites have work forces that are only 
one deep at each position. This requires them to import replacement labor, incurring duplicate 
travel costs and double labor costs. Savings realized by allowing for online refresher training are 
quite striking for these facilities (Table 10, Table 11, Table 12, and Table 13). 

Table 10. DOCS Costs by Category for Single Source at Large Urban Source. 


i Direct Environmental Activity Process Cost 

Indirect Environmental 
Activity Costs 

Other Costs 

| Start-Up 

O&M 

Activity 

$ K 

Activity 

$ 

Activity 

$ 

Activity 

$ 

Equipment 

purchase 

3.0* 

Consumables 
and supplies 

0.075 

Compliance Audits 

0 

Overhead associated 
with process 

0 

Equipment 

design 

NA 

Equipment 

maintenance 

0.10 

Document 

Maintenance 

0.06 

Productivity/cycle 

time 

0.48** 

Training 

operators 

3.87 

Training of 

operators 

1.88 

Environmental 
management plan 
preparation and 
maintenance 

0.06 

Worker injury 
claims and health 

costs 

0 





Reporting 

requirements 

0.12 



TOTAL 

6.87 


2.055 


0.24 


0.48 


* Costs for an initial DOCS type system will be considerably less as a practical issue. Costs cited are for a dedicated computer to be 
purchased at $1,500. This will seldom be necessary. Rights to the software are cited at $500 per unit, but negotiation is proceeding at this 
time to provide the government with cost-free rights to use this technology. Current camera costs are much less than the $750 cited and are 
continually trending downward. 

** Replacement labor as cited may not always be needed at larger facilities since there is often enough redundancy in the labor force to cover 
for workers who are absent for training. 
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Table 11. DOCS Costs by Category for Single Source at Remote Site. 


Direct Environmental Activity Process Cost 

Indirect Environmental 
Activity Costs 

Other Costs 

| Start-Up 

O&M 

Activity 

$ K 

Activity 

$ 

Activity 

$ 

Activity 

$ 

Equipment 

purchase 

* 

© 

cn 

Consumables 
and supplies 

0.075 

Compliance audits 

0 

Overhead associated 
with process 

0 

Equipment 

design 

NA 

Equipment 

maintenance 

0.10 

Document 

maintenance 

0.06 

Productivity/cycle 

time 

0.48** 

Training 

operators 

6.06 

Training of 

operators 

1.88 

Environmental 
management plan 
preparation and 
maintenance 

0.06 

Worker injury 
claims and health 

costs 

0 





Reporting 

requirements 

0.12 



TOTAL 

9.06 


2.055 


0.24 


0.48 


* Costs for an initial DOCS type system will be considerably less as a practical issue. Costs cited are for a dedicated computer to be 
purchased at $1,500. This will seldom be necessary. Rights to the software are cited at $500 per unit, but negotiation is proceeding at this 
time to provide the government with cost-free rights to use this technology. Current camera costs are much less than the $750 cited and are 
continually trending downward. 

** Replacement labor as cited is probably not needed when using the digital system since recurring training will likely consist of, at most, 
computer-aided “refamiliarization” training. 


Table 12. EPA Method #9 Costs by Category for Single Source at Large Urban Source. 


| Direct Environmental Activity Process Cost 

Indirect Environmental 
Activity Costs 

Other Costs 

| Start-Up 

O&M 

Activity 

$ K 

Activity 

$ 

Activity 

$ 

Activity 

$ 

Equipment 

purchase 

0 

Consumables 
and supplies 

0.025 

Compliance audits 

0 

Overhead associated 
with process 

0 

Equipment 

design 

NA 

Equipment 

maintenance 

0 

Document 

maintenance 

0.06 

Productivity/cycle 

time 

2.4* 

Training 

operators 

4.1 

Training of 

operators 

8.2 

Environmental 
management plan 
preparation and 
maintenance 

0.06 

Worker injury 
claims and health 

costs 

0 





Reporting 

requirements 

0.12 



TOTAL 

4.1 


8.225 


0.24 


2.4 


Replacement labor as cited is probably not needed when using the digital system since recurring training will likely consist of, at most, 
computer-aided “refamiliarization” training. 


Table 13. EPA Method #9 Costs for Single Source at Remote Site. 


| Direct Environmental Activity Process Cost 

Indirect Environmental 
Activity Costs 

Other Costs 

| Start-Up 

O&M 

Activity 

$ K 

Activity 

$ 

Activity 

$ 

Activity 

$ 

Equipment 

purchase 

0 

Consumables 
and supplies 

0.025 

Compliance audits 

0 

Overhead associated 
with process 

0 

Equipment 

design 

NA 

Equipment 

maintenance 

0 

Document 

maintenance 

0.06 

Productivity/cycle 

time 

4.8 

Training 

operators 

6.3 

Training of 

Operators 

12.6 

Environmental 
management plan 
preparation and 
maintenance 

0.06 

Worker injury 
claims and health 

costs 

0 





Reporting 

requirements 

0.12 



TOTAL 

6.3 


12.625 


0.24 


4.8 
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5.2 COST ANALYSIS 


Smoke school costs used are actual costs for the smoke schools attended in this field work. 
Tuition costs are standard at $350 per class per student. Labor rates are averages for the Hill 
AFB work force. Travel is estimated to be the same for each worker regardless of the type 
training. Training time is less for digital smoke readers since they are educated in a classroom 
and will not be required to go through exercises designed to calibrate their impressions of plume 
opacity. 

Cameras and software used for this test were still usable after 3 years; however, there have been 
quantum leaps in the advancement of this technology. Better quality equipment is now available 
at much lower cost and we expect this trend to continue. Software availability is an issue at the 
time of this report since the original vendor of the DOCS technology has quit producing, 
marketing, and promoting the technology. Demand is high enough that Hill AFB is pursuing 
purchase of the patent for the technology for the Air Force and DoD and licensing it to private 
industry. This will make government use even less expensive and may result in cost savings and 
technology advances for the market in general. 

DOCS needs little or no maintenance. With proper cleaning and care the system should operate 
for at least 3 years, perhaps considerably longer. Cost of upgrades to hardware and software 
should be negligible. The system contains no inherent hazardous chemicals, and disposal of this 
system will require no unique disposal methodology. Our field experience indicates that 
occasional rechargeable batteries will need replacement as will data cards and disks, and some 
minor camera repair will be needed with on-hand technology. This would amount to no more 
than $ 175 per year per source. 

Consumables and supplies in support of Method #9 measurements are estimated to cost $25 per 
year. For digital opacity measurements the consumables costs are estimated to be approximately 
$75 per year as batteries and memory chips for the cameras will need replacement at times. 

There will be no equipment maintenance cost for Method #9 measurements and only nominal 
equipment maintenance costs for the digital systems, estimated at $100 per year per source. 

5.3 COST COMPARISON 

The potential cost benefits to employing digital opacity monitoring technology vary with the size 
and type of facility being regulated for stationary source opacity. On the one hand, large 
facilities such as Air Force logistics facilities, Navy rework facilities, and Anny depots have 
large work forces that are flexible enough to be able to cover for workers who are absent for 
training. Modest per-source savings, however, add up when considering the large number of 
sources at such facilities. (Hill AFB has more than 1,600 emissions sources and trains more than 
35 smoke readers twice each year.) At the other end of the size spectrum are remote facilities 
where only one or two sources are regulated, the work force is very small, and there is little 
organic ability to cover for workers absent for training. In such instances, i.e., remote sites in 
Alaska and elsewhere, replacement workers will always need to be scheduled and transported to 
the sites in time for the existing labor force to travel for training. This overlap in labor cost 
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accounts for the demonstrable cost advantages to employing a digital type system (Table 11 and 
Table 13). It is at facilities such as these where the benefits of the digital system are greatest. 

DOCS is more costly to install than training certified observers but comparison of the annual 
operating costs presented in Table 10 through Table 13 shows an annual saving of $8,090 using 
DOCS versus certified readers at a large urban facility and an annual saving of $14,890 at a 
remote facility. In both cases, the purchase cost of a DOCS system, which is continually 
trending downwards, would easily be paid back within a year. 

Operator training for certified smoke readers currently would cost $4,100 for a large facility and 
$6,300 for remote site training. We anticipate that similar training for camera operations would 
cost $3,870 and $6,060, respectively, since it is anticipated that initial training will be 1 day 
shorter. These costs include: 

• Recurring training of operators will differ considerably since certified readers will need 
to attend smoke school every 6 months. (Annual cost would be $12,600 for remote 
sources and $8,200 for larger facilities.) The method of retraining for digital operators is 
anticipated to be computer generated training for recurring familiarization of the camera 
operators. They will not need to “calibrate” their eyes as certified readers would. Costs 
are estimated to be $1,880 for both large and remote facilities. 

• There is no expectation that there will be any difference in indirect environmental costs 
regardless of the method of opacity chosen. 

• We anticipate no costs associated with overhead or worker injury claims, regardless of 
which process is employed to measure opacity. Productivity/cycle time costs will vary 
with the process employed, as lost productivity associated with employees absent for 
training needs to be accounted for. At large bases the productivity lost for recurring 
training of smoke readers is anticipated to be $2,400 (2 individuals, 40 hrs at $ 15/hr twice 
a year). This will be doubled for remote sites, as replacement workers will be employed. 
Digital productivity losses will possibly be for the 8 hrs that employees spend retraining 
at the computer, $480 per source. 
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6.0 IMPLEMENTATION ISSUES 


6.1 ENVIRONMENTAL CHECKLIST 

This system falls solely under EPA Method #9. DOCS will be used only at facilities currently 
required to observe smoke stack opacity pursuant to this method. There will be no adverse 
environmental impacts associated with this technology. 

6.2 OTHER REGULATORY ISSUES 

EPA Reference Method #9 remains the standard reference method as per the CFR. To be 
considered an alternative to EPA Reference Method #9, language describing it as an alternative 
will need to be inserted into the CFR. The EPA EMC, which is the gatekeeper for this type of 
action, is currently monitoring the preliminary results from the DOCS 1-year regulatory pilot 
tests being conducted in Alaska, Texas, and Utah before proceeding with the development of an 
alternative test method. EMC agrees that the DOCS technology has utility and they are willing 
to monitor progress toward developing the technology further. They strongly recommend, 
however, that before EPA can move forward in the method approval process, the DOCS 1-year 
regulatory pilot study must collect a sufficient amount and type of field opacity data under a 
range of regulatory inspection conditions to demonstrate that the technology meets the credible 
evidence standard. 

The Air Force Research Laboratory (AFRL) will support the conduct of the 1-year credible 
evidence study; however, a new lead organization such as the EPA itself will need to be the 
program management element for such a study. Alaska, Kentucky, Tennessee, Texas, and Utah 
are in general support of this credible evidence 1-year test. The long-term potential for this 
technology and its general use may hinge on market forces rather than EPA rulemaking; 
however, initial interest in the general approach is universally high. Should enough credible 
evidence be gathered to demonstrate utility of the technology, then the permitting process within 
individual states can be worked via the Title V process and—potentially, at least—digital 
technology could be interwoven into each SIP. Limited use then will follow and after time, as 
use increases, the EPA EMC would have the bottom-up support it needs to consider adopting it 
as an alternative method. 

The AFRL has been approached to develop this technology as a means of measuring 
fugitive/nuisance dusts. There is some need for such technology within the Air Force, 
particularly on training and testing ranges, but it is felt that such a project will be better served if 
it were jointly led by both the U.S. Army and the U.S. Air Force, particularly since the U.S. 
Army has the lead between the services for range R&D. A follow-on ESTCP project proposal 
focused on the use of the DOCS technology for quantifying visible opacity associated with 
fugitive dusts was recently submitted, with the U.S. Army and U.S. Air Force listed as co-leads 
together with EMC, EPA Regions 6 and 8 as well as the state regulatory agencies of Alaska, 
California, Kentucky, Tennessee, Texas, and Utah identified as project collaborators. 
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6.3 END-USER ISSUES 


Tech transfer should be relatively straightforward. The hardware associated with DOCS is 
COTS technology and easily procured. Algorithms for the software are proprietary and have 
been patented. Hill AFB is investigating the potential for purchasing the patent for the 
technology, advertising for potential clients to license the technology for commercial 
distribution, and providing it free for DOD use. 
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